Project Overview and Import Libraries

In [1]:
Collecting pip
  Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 10.9MB/s ta 0:00:01    73% |███████████████████████▍        | 1.1MB 19.3MB/s eta 0:00:01
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-21.0.1
Collecting plotly==4.5.0
  Downloading plotly-4.5.0-py2.py3-none-any.whl (7.1 MB)
     |████████████████████████████████| 7.1 MB 5.7 MB/s eta 0:00:01     |██████████                      | 2.2 MB 5.7 MB/s eta 0:00:01     |███████████████████▊            | 4.4 MB 5.7 MB/s eta 0:00:01     |████████████████████████████████| 7.1 MB 5.7 MB/s 
Collecting retrying>=1.3.3
  Downloading retrying-1.3.3.tar.gz (10 kB)
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from plotly==4.5.0) (1.11.0)
Building wheels for collected packages: retrying
  Building wheel for retrying (setup.py) ... done
  Created wheel for retrying: filename=retrying-1.3.3-py3-none-any.whl size=9566 sha256=c3ca3955edee65d29fcd2795743ffd60001aa1c0389f2d9cf50c16433df6a769
  Stored in directory: /root/.cache/pip/wheels/ac/cb/8a/b27bf6323e2f4c462dcbf77d70b7c5e7868a7fbe12871770cf
Successfully built retrying
Installing collected packages: retrying, plotly
  Attempting uninstall: plotly
    Found existing installation: plotly 2.0.15
    Uninstalling plotly-2.0.15:
      Successfully uninstalled plotly-2.0.15
Successfully installed plotly-4.5.0 retrying-1.3.3
Collecting dask==2020.12.0
  Downloading dask-2020.12.0-py3-none-any.whl (884 kB)
     |████████████████████████████████| 884 kB 6.7 MB/s eta 0:00:01
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.6/site-packages (from dask==2020.12.0) (3.12)
Installing collected packages: dask
  Attempting uninstall: dask
    Found existing installation: dask 0.16.1
    Uninstalling dask-0.16.1:
      Successfully uninstalled dask-0.16.1
Successfully installed dask-2020.12.0
Collecting widgetsnbextension==3.0.0
  Downloading widgetsnbextension-3.0.0-py2.py3-none-any.whl (2.5 MB)
     |████████████████████████████████| 2.5 MB 5.3 MB/s eta 0:00:01
Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.6/site-packages (from widgetsnbextension==3.0.0) (5.7.0)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (2.10)
Requirement already satisfied: tornado>=4 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (4.5.3)
Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (17.1.2)
Requirement already satisfied: ipython_genutils in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (0.2.0)
Requirement already satisfied: traitlets>=4.2.1 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (4.3.2)
Requirement already satisfied: jupyter_core>=4.4.0 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (4.4.0)
Requirement already satisfied: jupyter_client>=5.2.0 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (5.2.4)
Requirement already satisfied: nbformat in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (4.4.0)
Requirement already satisfied: nbconvert in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (5.4.0)
Requirement already satisfied: ipykernel in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (4.9.0)
Requirement already satisfied: Send2Trash in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (1.5.0)
Requirement already satisfied: terminado>=0.8.1 in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (0.8.1)
Requirement already satisfied: prometheus_client in /opt/conda/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension==3.0.0) (0.3.1)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.6/site-packages (from jupyter_client>=5.2.0->notebook>=4.4.1->widgetsnbextension==3.0.0) (2.6.1)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil>=2.1->jupyter_client>=5.2.0->notebook>=4.4.1->widgetsnbextension==3.0.0) (1.11.0)
Requirement already satisfied: decorator in /opt/conda/lib/python3.6/site-packages (from traitlets>=4.2.1->notebook>=4.4.1->widgetsnbextension==3.0.0) (4.0.11)
Requirement already satisfied: ipython>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (6.5.0)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.15 in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (1.0.15)
Requirement already satisfied: pexpect in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (4.3.1)
Requirement already satisfied: pygments in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (2.2.0)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.7.4)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (38.4.0)
Requirement already satisfied: simplegeneric>0.8 in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.8.1)
Requirement already satisfied: backcall in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.1.0)
Requirement already satisfied: jedi>=0.10 in /opt/conda/lib/python3.6/site-packages (from ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.10.2)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.1.7)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension==3.0.0) (1.0)
Requirement already satisfied: mistune>=0.8.1 in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.8.3)
Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.2.3)
Requirement already satisfied: bleach in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (1.5.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (1.4.1)
Requirement already satisfied: testpath in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.3.1)
Requirement already satisfied: defusedxml in /opt/conda/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.5.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.6/site-packages (from nbformat->notebook>=4.4.1->widgetsnbextension==3.0.0) (2.6.0)
Requirement already satisfied: html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999 in /opt/conda/lib/python3.6/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.9999999)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.6/site-packages (from pexpect->ipython>=4.0.0->ipykernel->notebook>=4.4.1->widgetsnbextension==3.0.0) (0.5.2)
Installing collected packages: widgetsnbextension
  Attempting uninstall: widgetsnbextension
    Found existing installation: widgetsnbextension 3.1.0
    Uninstalling widgetsnbextension-3.1.0:
      Successfully uninstalled widgetsnbextension-3.1.0
Successfully installed widgetsnbextension-3.0.0
Collecting jsonschema==3.0.1
  Downloading jsonschema-3.0.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 2.1 MB/s eta 0:00:011
Requirement already satisfied: six>=1.11.0 in /opt/conda/lib/python3.6/site-packages (from jsonschema==3.0.1) (1.11.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.6/site-packages (from jsonschema==3.0.1) (0.15.2)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from jsonschema==3.0.1) (38.4.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from jsonschema==3.0.1) (19.1.0)
Installing collected packages: jsonschema
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 2.6.0
    Uninstalling jsonschema-2.6.0:
      Successfully uninstalled jsonschema-2.6.0
Successfully installed jsonschema-3.0.1
Collecting tensorflow==2.3.2
  Downloading tensorflow-2.3.2-cp36-cp36m-manylinux2010_x86_64.whl (320.4 MB)
     |███████████████████▉            | 198.4 MB 29.8 MB/s eta 0:00:05  |                                | 665 kB 6.5 MB/s eta 0:00:49     |▋                               | 6.6 MB 6.5 MB/s eta 0:00:49     |▉                               | 8.3 MB 6.5 MB/s eta 0:00:48     |█▋                              | 16.1 MB 6.5 MB/s eta 0:00:47     |█▊                              | 17.3 MB 6.5 MB/s eta 0:00:47     |██▎                             | 23.3 MB 28.6 MB/s eta 0:00:11     |██▍                             | 23.8 MB 28.6 MB/s eta 0:00:11     |██▌                             | 25.2 MB 28.6 MB/s eta 0:00:11     |██▋                             | 26.6 MB 28.6 MB/s eta 0:00:11     |███                             | 29.4 MB 28.6 MB/s eta 0:00:11     |███                             | 30.8 MB 28.6 MB/s eta 0:00:11     |███▌                            | 34.8 MB 28.6 MB/s eta 0:00:10     |███▋                            | 36.2 MB 28.6 MB/s eta 0:00:10     |███▊                            | 37.6 MB 28.6 MB/s eta 0:00:10     |███▉                            | 38.9 MB 28.6 MB/s eta 0:00:10     |████                            | 40.1 MB 28.6 MB/s eta 0:00:10     |████▏                           | 41.5 MB 28.6 MB/s eta 0:00:10     |████▎                           | 42.9 MB 28.6 MB/s eta 0:00:10     |████▋                           | 46.5 MB 28.5 MB/s eta 0:00:10     |████▊                           | 47.2 MB 28.5 MB/s eta 0:00:10     |█████                           | 50.8 MB 28.5 MB/s eta 0:00:10     |█████▏                          | 52.0 MB 28.5 MB/s eta 0:00:10     |█████▍                          | 54.4 MB 28.5 MB/s eta 0:00:10     |█████▉                          | 58.1 MB 28.5 MB/s eta 0:00:10     |██████                          | 59.2 MB 28.5 MB/s eta 0:00:10     |██████▎                         | 62.8 MB 28.5 MB/s eta 0:00:10     |██████▍                         | 64.0 MB 28.5 MB/s eta 0:00:10     |██████▌                         | 65.3 MB 28.5 MB/s eta 0:00:09     |██████▋                         | 66.5 MB 28.5 MB/s eta 0:00:09     |███████                         | 69.0 MB 26.7 MB/s eta 0:00:10     |███████▏                        | 71.4 MB 26.7 MB/s eta 0:00:10     |███████▎                        | 72.7 MB 26.7 MB/s eta 0:00:10     |███████▌                        | 75.1 MB 26.7 MB/s eta 0:00:10     |███████▋                        | 76.5 MB 26.7 MB/s eta 0:00:10     |███████▉                        | 78.2 MB 26.7 MB/s eta 0:00:10     |████████                        | 79.4 MB 26.7 MB/s eta 0:00:10     |████████                        | 80.8 MB 26.7 MB/s eta 0:00:09     |████████▏                       | 82.3 MB 26.7 MB/s eta 0:00:09     |████████▉                       | 88.5 MB 26.7 MB/s eta 0:00:09     |█████████▏                      | 92.1 MB 19.9 MB/s eta 0:00:12     |█████████▍                      | 94.1 MB 19.9 MB/s eta 0:00:12     |█████████▋                      | 95.8 MB 19.9 MB/s eta 0:00:12     |██████████▏                     | 102.1 MB 19.9 MB/s eta 0:00:11     |██████████▌                     | 104.6 MB 19.9 MB/s eta 0:00:11     |██████████▋                     | 105.7 MB 19.9 MB/s eta 0:00:11     |██████████▊                     | 107.2 MB 19.9 MB/s eta 0:00:11     |███████████                     | 110.1 MB 19.9 MB/s eta 0:00:11     |███████████▏                    | 111.3 MB 19.9 MB/s eta 0:00:11     |███████████▎                    | 112.5 MB 22.5 MB/s eta 0:00:10     |███████████▍                    | 113.6 MB 22.5 MB/s eta 0:00:10     |███████████▌                    | 115.3 MB 22.5 MB/s eta 0:00:10     |███████████▊                    | 117.0 MB 22.5 MB/s eta 0:00:10     |███████████▉                    | 118.7 MB 22.5 MB/s eta 0:00:09     |████████████                    | 120.0 MB 22.5 MB/s eta 0:00:09     |████████████▏                   | 121.7 MB 22.5 MB/s eta 0:00:09     |████████████▎                   | 123.3 MB 22.5 MB/s eta 0:00:09     |████████████▋                   | 126.2 MB 22.5 MB/s eta 0:00:09     |████████████▊                   | 126.9 MB 22.5 MB/s eta 0:00:09     |████████████▉                   | 128.0 MB 22.5 MB/s eta 0:00:09     |█████████████                   | 129.6 MB 22.5 MB/s eta 0:00:09     |█████████████▎                  | 132.5 MB 22.5 MB/s eta 0:00:09     |█████████████▍                  | 134.2 MB 22.5 MB/s eta 0:00:09     |█████████████▌                  | 135.7 MB 22.5 MB/s eta 0:00:09     |█████████████▊                  | 137.3 MB 22.5 MB/s eta 0:00:09     |██████████████                  | 139.2 MB 19.2 MB/s eta 0:00:10     |██████████████                  | 140.8 MB 19.2 MB/s eta 0:00:10     |██████████████▎                 | 142.5 MB 19.2 MB/s eta 0:00:10     |██████████████▍                 | 144.3 MB 19.2 MB/s eta 0:00:10     |██████████████▋                 | 145.9 MB 19.2 MB/s eta 0:00:10     |██████████████▊                 | 147.1 MB 19.2 MB/s eta 0:00:10     |██████████████▉                 | 148.7 MB 19.2 MB/s eta 0:00:09     |███████████████                 | 150.3 MB 19.2 MB/s eta 0:00:09     |███████████████▏                | 151.5 MB 19.2 MB/s eta 0:00:09     |███████████████▎                | 152.8 MB 19.2 MB/s eta 0:00:09     |███████████████▍                | 153.9 MB 19.2 MB/s eta 0:00:09     |███████████████▌                | 155.5 MB 19.2 MB/s eta 0:00:09     |███████████████▊                | 156.9 MB 19.2 MB/s eta 0:00:09     |████████████████                | 160.1 MB 19.2 MB/s eta 0:00:09     |████████████████▏               | 161.4 MB 19.2 MB/s eta 0:00:09     |████████████████▎               | 162.7 MB 19.2 MB/s eta 0:00:09     |████████████████▍               | 164.4 MB 31.5 MB/s eta 0:00:05     |████████████████▋               | 166.2 MB 31.5 MB/s eta 0:00:05     |████████████████▊               | 167.5 MB 31.5 MB/s eta 0:00:05     |█████████████████               | 170.6 MB 31.5 MB/s eta 0:00:05     |█████████████████▏              | 171.6 MB 31.5 MB/s eta 0:00:05     |█████████████████▍              | 174.3 MB 31.5 MB/s eta 0:00:05     |█████████████████▋              | 176.0 MB 31.5 MB/s eta 0:00:05     |█████████████████▊              | 177.4 MB 31.5 MB/s eta 0:00:05     |█████████████████▉              | 178.6 MB 31.5 MB/s eta 0:00:05     |██████████████████              | 180.3 MB 31.5 MB/s eta 0:00:05     |██████████████████▏             | 181.7 MB 31.5 MB/s eta 0:00:05     |██████████████████▎             | 183.2 MB 31.5 MB/s eta 0:00:05     |██████████████████▋             | 186.0 MB 31.5 MB/s eta 0:00:05     |██████████████████▊             | 187.4 MB 31.5 MB/s eta 0:00:05     |██████████████████▉             | 188.7 MB 31.5 MB/s eta 0:00:05     |███████████████████             | 190.5 MB 29.8 MB/s eta 0:00:05     |███████████████████▏            | 192.0 MB 29.8 MB/s eta 0:00:05     |███████████████████▍            | 193.6 MB 29.8 MB/s eta 0:00:05     |███████████████████▌            | 195.1 MB 29.8 MB/s eta 0:00:05     |███████████████████▊            | 197.1 MB 29.8 MB/s eta 0:00:05
     |████████████████████            | 199.8 MB 29.8 MB/s eta 0:00:05
     |████████████████████            | 201.1 MB 29.8 MB/s eta 0:00:04
     |████████████████████▏           | 202.4 MB 29.8 MB/s eta 0:00:04
     |████████████████████▍           | 203.7 MB 29.8 MB/s eta 0:00:04
     |████████████████████████████████| 320.4 MB 3.3 kB/s  eta 0:00:01     |████████████████████▌           | 205.7 MB 29.8 MB/s eta 0:00:04     |████████████████████▋           | 206.8 MB 29.8 MB/s eta 0:00:04     |████████████████████▉           | 208.5 MB 29.8 MB/s eta 0:00:04��█████████████           | 210.4 MB 29.8 MB/s eta 0:00:04     |█████████████████████▏          | 212.0 MB 29.8 MB/s eta 0:00:04     |█████████████████████▎          | 213.4 MB 29.8 MB/s eta 0:00:04     |█████████████████████▌          | 214.8 MB 10.8 MB/s eta 0:00:10     |█████████████████████▋          | 216.2 MB 10.8 MB/s eta 0:00:10     |█████████████████████▊          | 217.3 MB 10.8 MB/s eta 0:00:10     |█████████████████████▉          | 219.0 MB 10.8 MB/s eta 0:00:10     |██████████████████████          | 220.6 MB 10.8 MB/s eta 0:00:10     |██████████████████████▏         | 222.1 MB 10.8 MB/s eta 0:00:10     |██████████████████████▍         | 223.7 MB 10.8 MB/s eta 0:00:09     |██████████████████████▊         | 227.2 MB 10.8 MB/s eta 0:00:09     |███████████████████████         | 229.2 MB 10.8 MB/s eta 0:00:09     |███████████████████████         | 230.1 MB 10.8 MB/s eta 0:00:09███████▏        | 231.5 MB 10.8 MB/s eta 0:00:09█████████▎        | 232.7 MB 10.8 MB/s eta 0:00:09█████████▍        | 234.6 MB 10.8 MB/s eta 0:00:08     |███████████████████████▋        | 236.0 MB 10.8 MB/s eta 0:00:08     |███████████████████████▊        | 237.9 MB 10.8 MB/s eta 0:00:08     |████████████████████████▏       | 241.4 MB 17.9 MB/s eta 0:00:05     |████████████████████████▏       | 242.5 MB 17.9 MB/s eta 0:00:05     |████████████████████████▍       | 244.0 MB 17.9 MB/s eta 0:00:05��████████████████████▌       | 245.5 MB 17.9 MB/s eta 0:00:05     |████████████████████████▊       | 247.1 MB 17.9 MB/s eta 0:00:05     |████████████████████████▉       | 248.7 MB 17.9 MB/s eta 0:00:05     |█████████████████████████       | 250.6 MB 17.9 MB/s eta 0:00:04     |█████████████████████████▏      | 252.4 MB 17.9 MB/s eta 0:00:04     |█████████████████████████▋      | 256.2 MB 17.9 MB/s eta 0:00:04     |█████████████████████████▊      | 257.9 MB 17.9 MB/s eta 0:00:04     |█████████████████████████▉      | 259.2 MB 17.9 MB/s eta 0:00:04     |██████████████████████████      | 260.8 MB 17.9 MB/s eta 0:00:04     |██████████████████████████▏     | 262.1 MB 17.9 MB/s eta 0:00:04��██▍     | 263.8 MB 17.9 MB/s eta 0:00:04�█▌     | 265.1 MB 37.1 MB/s eta 0:00:02     |██████████████████████████▋     | 266.9 MB 37.1 MB/s eta 0:00:02     |██████████████████████████▉     | 268.2 MB 37.1 MB/s eta 0:00:02��███████     | 269.7 MB 37.1 MB/s eta 0:00:021.2 MB 37.1 MB/s eta 0:00:02:00:02�██████████████████████▍    | 274.5 MB 37.1 MB/s eta 0:00:02�████████████▋    | 276.2 MB 37.1 MB/s eta 0:00:02�██▊    | 278.0 MB 37.1 MB/s eta 0:00:02�█████████████    | 279.9 MB 37.1 MB/s eta 0:00:02     |████████████████████████████    | 281.3 MB 37.1 MB/s eta 0:00:02     |████████████████████████████▏   | 282.4 MB 37.1 MB/s eta 0:00:02�███████████████████████▍   | 284.1 MB 37.1 MB/s eta 0:00:01     |████████████████████████████▌   | 285.1 MB 37.1 MB/s eta 0:00:01     |████████████████████████████▋   | 286.8 MB 37.1 MB/s eta 0:00:01��████████████████████▉   | 288.4 MB 37.1 MB/s eta 0:00:01███████████████████████   | 290.4 MB 20.6 MB/s eta 0:00:02:00:02�██████████████████████████▍  | 293.8 MB 20.6 MB/s eta 0:00:02�████▌  | 295.2 MB 20.6 MB/s eta 0:00:028 MB 20.6 MB/s eta 0:00:02 |█████████████████████████████▉  | 298.4 MB 20.6 MB/s eta 0:00:02▏ | 302.4 MB 20.6 MB/s eta 0:00:01��██████████▍ | 304.1 MB 20.6 MB/s eta 0:00:010:01��██████████████████████▋ | 306.8 MB 20.6 MB/s eta 0:00:01��██████████████████████▉ | 309.0 MB 20.6 MB/s eta 0:00:01�██████████████████████ | 310.8 MB 20.6 MB/s eta 0:00:01�██▏| 312.1 MB 20.6 MB/s eta 0:00:01��███████▍| 314.5 MB 20.6 MB/s eta 0:00:010:01�██████████▉| 318.6 MB 10.8 MB/s eta 0:00:01██████████████| 319.7 MB 10.8 MB/s eta 0:00:01
Collecting wrapt>=1.11.1
  Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting absl-py>=0.7.0
  Downloading absl_py-0.12.0-py3-none-any.whl (129 kB)
     |████████████████████████████████| 129 kB 11.3 MB/s eta 0:00:01
Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.6/site-packages (from tensorflow==2.3.2) (0.30.0)
Collecting six>=1.12.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting tensorboard<3,>=2.3.0
  Downloading tensorboard-2.5.0-py3-none-any.whl (6.0 MB)
     |████████████████████████████████| 6.0 MB 12.4 MB/s eta 0:00:01   |                                | 20 kB 5.0 MB/s eta 0:00:02     |███████████▋                    | 2.2 MB 12.4 MB/s eta 0:00:01     |█████████████████████▉          | 4.1 MB 12.4 MB/s eta 0:00:01��█████▉  | 5.6 MB 12.4 MB/s eta 0:00:01
Collecting grpcio>=1.8.6
  Downloading grpcio-1.37.0-cp36-cp36m-manylinux2014_x86_64.whl (4.2 MB)
     |████████████████████████████████| 4.2 MB 12.1 MB/s eta 0:00:01�██████▉              | 2.3 MB 12.1 MB/s eta 0:00:01
Collecting protobuf>=3.9.2
  Downloading protobuf-3.15.8-cp36-cp36m-manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 14.1 MB/s eta 0:00:01
Collecting termcolor>=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting google-pasta>=0.1.8
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     |████████████████████████████▌   | 51 kB 7.3 MB/s eta 0:00:01     |████████████████████████████████| 57 kB 3.3 MB/s 
Collecting keras-preprocessing<1.2,>=1.1.1
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 1.1 MB/s eta 0:00:01
Collecting opt-einsum>=2.3.2
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 2.9 MB/s  eta 0:00:01
Collecting h5py<2.11.0,>=2.10.0
  Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 12.7 MB/s eta 0:00:01     |█████▌                          | 491 kB 12.7 MB/s eta 0:00:01��██████████████████████▌       | 2.2 MB 12.7 MB/s eta 0:00:01
Collecting numpy<1.19.0,>=1.16.0
  Downloading numpy-1.18.5-cp36-cp36m-manylinux1_x86_64.whl (20.1 MB)
     |████████████████████████████████| 20.1 MB 9.9 MB/s eta 0:00:01    |██▉                             | 1.8 MB 9.9 MB/s eta 0:00:022     |████████████▌                   | 7.9 MB 9.9 MB/s eta 0:00:02 | 9.3 MB 9.9 MB/s eta 0:00:02��██████▎             | 11.5 MB 9.9 MB/s eta 0:00:01��█████████████████           | 13.1 MB 9.9 MB/s eta 0:00:01ta 0:00:01[K     |███████████████████████████▌    | 17.3 MB 9.9 MB/s eta 0:00:01��████████▌ | 19.2 MB 9.9 MB/s eta 0:00:01
Collecting tensorflow-estimator<2.4.0,>=2.3.0
  Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
     |████████████████████████████████| 459 kB 21.5 MB/s eta 0:00:01
Collecting astunparse==1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.2) (0.14.1)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.4-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.2) (2.6.9)
Collecting requests<3,>=2.21.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 4.1 MB/s eta 0:00:01
Collecting setuptools>=41.0.0
  Downloading setuptools-56.0.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 17.6 MB/s eta 0:00:01
Collecting google-auth<2,>=1.6.3
  Downloading google_auth-1.29.0-py2.py3-none-any.whl (142 kB)
     |████████████████████████████████| 142 kB 20.0 MB/s eta 0:00:01
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
     |████████████████████████████████| 781 kB 18.6 MB/s eta 0:00:01�████████▌                     | 256 kB 18.6 MB/s eta 0:00:01
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.0-py3-none-manylinux2010_x86_64.whl (3.9 MB)
     |████████████████████████████████| 3.9 MB 12.2 MB/s eta 0:00:01     |█████████████▍                  | 1.6 MB 12.2 MB/s eta 0:00:01��██████████████▉     | 3.3 MB 12.2 MB/s eta 0:00:01
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 10.7 MB/s eta 0:00:01
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.1-py3-none-any.whl (12 kB)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (3.4.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.6/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (0.8.0)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 4.4 MB/s  eta 0:00:01
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (2019.11.28)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (2.6)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (3.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (1.22)
Requirement already satisfied: oauthlib>=0.6.2 in /opt/conda/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.2) (2.0.6)
Building wheels for collected packages: termcolor, wrapt
  Building wheel for termcolor (setup.py) ... done
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=5711 sha256=284d04a29abb7735671d43e3290345fbf2f3d1f8fc9f5a97ca7a2bc1fa53ecfa
  Stored in directory: /root/.cache/pip/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=71786 sha256=d8542c9106780186e6cf8c5da3f8b667fb250fa48126f9c037779be2b931b160
  Stored in directory: /root/.cache/pip/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63
Successfully built termcolor wrapt
Installing collected packages: pyasn1, six, setuptools, requests, pyasn1-modules, cachetools, google-auth, tensorboard-plugin-wit, tensorboard-data-server, protobuf, numpy, grpcio, google-auth-oauthlib, absl-py, wrapt, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, h5py, google-pasta, gast, astunparse, tensorflow
  Attempting uninstall: pyasn1
    Found existing installation: pyasn1 0.4.4
    Uninstalling pyasn1-0.4.4:
      Successfully uninstalled pyasn1-0.4.4
  Attempting uninstall: six
    Found existing installation: six 1.11.0
    Uninstalling six-1.11.0:
      Successfully uninstalled six-1.11.0
  Attempting uninstall: setuptools
    Found existing installation: setuptools 38.4.0
    Uninstalling setuptools-38.4.0:
      Successfully uninstalled setuptools-38.4.0
  Attempting uninstall: requests
    Found existing installation: requests 2.18.4
    Uninstalling requests-2.18.4:
      Successfully uninstalled requests-2.18.4
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.5.1
    Uninstalling protobuf-3.5.1:
      Successfully uninstalled protobuf-3.5.1
  Attempting uninstall: numpy
    Found existing installation: numpy 1.12.1
    Uninstalling numpy-1.12.1:
      Successfully uninstalled numpy-1.12.1
  Attempting uninstall: h5py
    Found existing installation: h5py 2.7.1
    Uninstalling h5py-2.7.1:
      Successfully uninstalled h5py-2.7.1
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 1.3.0
    Uninstalling tensorflow-1.3.0:
      Successfully uninstalled tensorflow-1.3.0
Successfully installed absl-py-0.12.0 astunparse-1.6.3 cachetools-4.2.1 gast-0.3.3 google-auth-1.29.0 google-auth-oauthlib-0.4.4 google-pasta-0.2.0 grpcio-1.37.0 h5py-2.10.0 keras-preprocessing-1.1.2 numpy-1.18.5 opt-einsum-3.3.0 protobuf-3.15.8 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.25.1 setuptools-56.0.0 six-1.15.0 tensorboard-2.5.0 tensorboard-data-server-0.6.0 tensorboard-plugin-wit-1.8.0 tensorflow-2.3.2 tensorflow-estimator-2.3.0 termcolor-1.1.0 wrapt-1.12.1
Collecting numpy==1.17.5
  Downloading numpy-1.17.5-cp36-cp36m-manylinux1_x86_64.whl (20.0 MB)
     |████████████████████████████████| 20.0 MB 227 kB/s eta 0:00:01    |████▎                           | 2.7 MB 5.5 MB/s eta 0:00:04     |██████▎                         | 3.9 MB 5.5 MB/s eta 0:00:03     |██████████████████████████████▊ | 19.2 MB 5.5 MB/s eta 0:00:01
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.18.5
    Uninstalling numpy-1.18.5:
      Successfully uninstalled numpy-1.18.5
Successfully installed numpy-1.17.5
Name: tensorflow
Version: 2.3.2
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /opt/conda/lib/python3.6/site-packages
Requires: tensorflow-estimator, tensorboard, six, gast, wrapt, h5py, astunparse, absl-py, termcolor, keras-preprocessing, google-pasta, numpy, wheel, grpcio, opt-einsum, protobuf
Required-by: 
In [2]:
In [3]:
In [4]:
Tensorflow version: 2.3.2
In [5]:
Out[5]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f346c6930f0>

Load and Inspect the data

In [6]:
In [7]:
In [8]:
Out[8]:
Date AdjClose
0 2000-01-03 98.146072
1 2000-01-04 94.307983
2 2000-01-05 94.476685
3 2000-01-06 92.958336
4 2000-01-07 98.356987
In [9]:

Data Pre-processing

In [10]:
(4227, 2) (1057, 2)
In [11]:
In [12]:

Temporalize Data and Create Training and Test Splits

In [13]:
(4197, 30, 1) (1027, 30, 1)
In [14]:
30 1

Build an LSTM Autoencoder

In [15]:
In [16]:
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm (LSTM)                  (None, 128)               66560     
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
repeat_vector (RepeatVector) (None, 30, 128)           0         
_________________________________________________________________
lstm_1 (LSTM)                (None, 30, 128)           131584    
_________________________________________________________________
dropout_1 (Dropout)          (None, 30, 128)           0         
_________________________________________________________________
time_distributed (TimeDistri (None, 30, 1)             129       
=================================================================
Total params: 198,273
Trainable params: 198,273
Non-trainable params: 0
_________________________________________________________________

Train the Autoencoder

In [17]:
In [18]:
Epoch 1/100
119/119 [==============================] - 10s 84ms/step - loss: 0.1117 - val_loss: 0.1626
Epoch 2/100
119/119 [==============================] - 9s 76ms/step - loss: 0.0804 - val_loss: 0.2987
Epoch 3/100
119/119 [==============================] - 9s 77ms/step - loss: 0.0863 - val_loss: 0.2685
Epoch 4/100
119/119 [==============================] - 9s 74ms/step - loss: 0.1408 - val_loss: 0.1076
Epoch 5/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0972 - val_loss: 0.0580
Epoch 6/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0711 - val_loss: 0.0622
Epoch 7/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0626 - val_loss: 0.1231
Epoch 8/100
119/119 [==============================] - 9s 75ms/step - loss: 0.0700 - val_loss: 0.0467
Epoch 9/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0713 - val_loss: 0.1331
Epoch 10/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0851 - val_loss: 0.0707
Epoch 11/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0560 - val_loss: 0.1480
Epoch 12/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0593 - val_loss: 0.0542
Epoch 13/100
119/119 [==============================] - 9s 74ms/step - loss: 0.0683 - val_loss: 0.0554

Plot Metrics and Evaluate the Model

If validation loss is lower than Training loss, it means the model is underfitting.

If validation loss is higher than Training loss, it means the model is overfititng

In [19]:
In [20]:
In [21]:
In [22]:
In [23]:
In [24]:
In [25]:
33/33 [==============================] - 1s 15ms/step - loss: 0.1449
Out[25]:
0.14493736624717712
In [26]:
Out[26]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f3458194f60>
In [27]:
Out[27]:
Error    0.481012
dtype: float64
In [28]:
In [29]:
In [30]:

Detecting Anomalies in the Stock prices

In [31]:
In [32]:
Out[32]:
Date AdjClose Close loss threshold anomaly
4257 2016-12-02 -1.674541 2.499809 0.627805 0.55 True
4258 2016-12-05 -1.644069 2.532275 0.619620 0.55 True
4259 2016-12-06 -1.627910 2.549492 0.621465 0.55 True
4260 2016-12-07 -1.560965 2.620816 0.625362 0.55 True
4261 2016-12-08 -1.548268 2.634343 0.642843 0.55 True
In [33]:
Out[33]:
Date AdjClose Close loss threshold anomaly
5279 2020-12-24 2.493057 6.940073 0.171638 0.55 False
5280 2020-12-28 2.572426 7.024635 0.172281 0.55 False
5281 2020-12-29 2.554648 7.005695 0.206493 0.55 False
5282 2020-12-30 2.567918 7.019833 0.219301 0.55 False
5283 2020-12-31 2.615239 7.070250 0.230472 0.55 False
In [34]:
In [35]:
Jan 2017Jul 2017Jan 2018Jul 2018Jan 2019Jul 2019Jan 2020Jul 202000.20.40.60.811.21.4
Test LossThreshold
In [36]:
Out[36]:
Date AdjClose Close loss threshold anomaly
4257 2016-12-02 -1.674541 2.499809 0.627805 0.55 True
4258 2016-12-05 -1.644069 2.532275 0.619620 0.55 True
4259 2016-12-06 -1.627910 2.549492 0.621465 0.55 True
4260 2016-12-07 -1.560965 2.620816 0.625362 0.55 True
4261 2016-12-08 -1.548268 2.634343 0.642843 0.55 True
In [37]:
In [38]:
Jan 2017Jul 2017Jan 2018Jul 2018Jan 2019Jul 2019Jan 2020Jul 2020406080100120140160180200
AdjClose PriceAnomaly

As it was well documented that the stock hit a 2020 low in March owing to COVID-19 pandemic uncertainities, but quickly reaccelerated to a high point later on bullish expectations with coronavirus vaccine news and lock-down relaxations.

In [ ]: